“What Is Better Than Chi-Square?” and Related Koans

A koan is an enigmatic, and often senseless, question posed as an aid to meditation in Zen Buddhism. By “chi-square”, we mean Pearson’s general scheme of a p-value, or tail, statistic that is constructed as a sum of independent random variables, each of which is the square of a normal deviate N(0, 1) (or at least approximately so). Such a statistic is distributed according to the familiar χ2I distribution, with I degress of freedom, whose tail values are tabulated or readily computable. Chi-square tests are used to distinguish between two samples, under the null hypothesis that they are drawn from the same distribution. Generally, each sample has a number of subsamples or “bins” (i = 1, . . . , I) that are more or less independent, and the chi-square statistic is a sum over the bins, with one squared normal deviate (∼ χ21) obtained from each bin. If there are a small number of linear constraints among the values in the bins, so that they are not independent, then it is well known [6] that the I in χ2I is reduced by the number of constraints, with the general scheme otherwise unchanged. In this note, we are interested in the particular regime where the number of bins I is very large, and where the data in the bins are integer numbers of counts, mi for the first sample and ni for the second. Further, our interest is when the total numbers of counts is much larger than the number in any single bin, i.e., M ≡ ∑